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ABSTRACT 

This  paper  presents  a new  extremal  approach  to  deriving  dual 

optimization  problems  with  proper  duality  inequality  which  simplifies 

and  generalizes  the  Fenchel-Rockafellar  scheme.  Our  derivation  proceeds 

in  two  stages,  (i)  inequality  attainment,  (ii)  decoupling  primal  and  dual 

variables.  The  power  and  convenience  of  this  approach  are  exhibited 

through  a new,  much  simpler  derivation  of  the  Charnes-Cooper  resuits 

for  Khinchin-Kullback-Leibler  statistical  estimation  [1],  the  immediate 

2 

establishment  of  the  C duality  for  general  distributions  and  its  extensions 
to  general  linear  inequality  constraints,  plus  the  development  of  a new 
two-person  zero-sum  game  connection. 


0.  Introduction 


In  [1]  Charnes  and  Cooper  established  a duality  with  three  mutually 
exclusive  and  collectively  exhaustive  (MECE)  duality  states  for  constrained 
K2L  (Khinchin-Kullback-Leibler)  statistical  estimation  with  finite  discrete 
distributions.  Their  dual  problems  are 
(I)  max  v(6)  s 

rp 

with  = b , 6 > 0 

and 

(II)  inf  £(z)  = cTe  Az  - b*z 

Extensions  to  general  distributions  were  developed  in  [2]  by  Ben-Tal  and 
Charnes  using  function  space  settings  and  some  generalized  Fenchel-Rockefellar 
duality  theory  [3).  The  "F-R*' theory  was  also  employed  by  Ben-Tal  and 
Charnes  infi]  to  develop  a general  class  of  non-linear  inequality  constrained 
convex  programming  problems  with  unconstrained  duals  as  in  the  above 
”C2"  duality. 

This  paper  presents  a new  extremal  approach  to  deriving  dual  optimiza- 
tion problems  with  proper  duality  inequality  which  simplifies  and  generalizes 
the  Fenchel-Rockafellar  scheme.  Our  derivation  proceeds  in  two  stages, 

(i)  inequality  attainment,  (ii)  decoupling  primal  and  dual  variables.  The  power 
and  convenience  of  this  approach  are  exhibited  through  a new,  much  simpler 

derivation  of  the  Charnes -Cooper  results  for  Khinchin-Kullback-Leibler 

2 

statistical  estimation  [1],  the  immediate  establishment  of  the  C duality  for 
general  distributions  and  its  extensions  to  general  linear  inequality  constraints, 
plus  the  development  of  a new  two-person  zero-sum  game  connection. 

The  latter  dualities  as  well  as  extensions  to  general  classes  of  convex 
functionals  a la  [4]  are  being  developed  elsewhere  by  Charnes,  Cooper, 


Seiford  and  Palacios. 


: 
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1.  The  Extremal  Principle 

We  restrict  the  presentation  here  to  convex  programming  dualities. 

(The  general  case  involves  only  the  obvious  deletion  of  the  convexity  (concavity) 
requirements. ) There  are  two  aspects  to  the  principle.  The  first  is  the  achieve- 
ment of  the  duality  inequality.  The  second  is  "decoupling"  the  variables 
of  the  dual  problems. 

Let  K(6,  x)  be  a real-valued  function  concave  in  6 for  (6,  x)  e 
A(x)  X £Rm  x R S . This  function  is  chosen  so  that  g(6)  a inf  K(6,  x)  exists 
for  each  6 e A . (Thereby  g(6)  is  a concave  function  of  fi). 

For  convexity  re  x and  then  decoupling,  let  x = T(z)  map  Z,  a convex 
set  in  Rn,  into  X.  Let  K(  6,x)  = f(z),  a convex  function,  for  z e Z,  6 e P. 

Thus  P ©T(Z)  is  the  decoupling  set  for  (6,x).  For  convexity  in  6 we 
require  A to  be  a convex  set. 

Clearly,  then,  the  duality  inequality  is  established 
(1)  g(6)«f(z),  V A t AO  P,  V zeZ 

Further,  the  problems 

(III)  sup  g(6),  6 c Alir 

and 

(IV)  inf  f(  z),  z c Z 

are  dual  convex  programming  problems.  Existence  or  nonexistence  of 
duality  gaps  is  clearly  dependent  on  the  choices  of  AAPand  Z. 


l 
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2.  The  Dual  Problems  for  Discrete  K2-L  Estimation 
Consider  the  function  of  two  real  variables  6,  x, 

(2)  K (6,  x)  H cex  - 6x,  where  c,  6 > 0. 

Then  for  c > 0 and  6 i0, 

(2.  1)  g(  4)  = inf  K(6,x)  = -60r  (ec),  where  e is  the  base  of  the  Napierian 
logarithms,  and  we  set  0 • 9m  (0)  = 0. 

Note  that  if  c = 0 with  6 * 0,  then  we  must  restrict  x above,  e.g. 
x s 0,  in  order  that  g(6)  exist. 

Staying  with  c^  > 0 and  introducing  Xj  s 0,  we  have 


(2 


.2)  S (CieXi  - 6ixi), 


for  all  5'i0  and  all  x. 

Next,  suppose 

(2.  3)  x.  = h a^  zj  and  p 4jajj  = bj, 

for  constants  ajj  and  bj. 

Then,  in  vector-matrix  notation, 

(2.4)  v(4)  s£(z)  ■c^e'^z  ~ b^x 
T T 

for  6 > 0,  6A=b  and  z unconstrained. 

Thus  (I)  and  (II)  are  established  as  dual  convex  programs  satisfying  the 
duality  inequality  (2.4). 

Further,  they  are  established  for  the  general  discrete  distribution 


case  where  the  tx^}  1 £ z j } are  sequences  for  which  the  sums  in  (2.2), 
(2.3)  exist.  For  example,  we  could  have  x,  6,  z in  the  sequence  space 
(*l)  and, thereby,  A a linear  transformation  from  (4^)  into  Uj). 
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3.  The  Charnes-Cooper  Duality  States 

T T 

Clearly,  from  (2.4),  £(z)  is  bounded  below  provided  that  6 iO,  6 A = b 
has  a solution.  Suppose  it  does  not.  Consider  the  dual  l.p.  problems: 


(3.  1)  max  b 1 z 

AztO 


and  min  6 


T . 0 


«ta  = bT 


By  the  extended  dual  theorem  f 5]  , since  z = 0 satisfies  Az  s 0,  there  exists 

a sequence  zn  such  that  Azn  * 0 and  bTzn  -♦  00  . But  then, 

jp  (zn)  = c^e  ^ z - b^zn  £ c^e  -b^z11  ® 

Thus  we  have  established  duality  state  1 e.g. : 

nr  T1 

inf  l£(z)  = -»  iff  6 4 0,  6 A = b has  no  solution. 

Next,  suppose  ^(z)  has  a minimum.  Since  ^(z)  is  differentiable  and 
convex  aid  z is  unconstrained,  the  gradient  must  vanish  at  z*,  a minimum,  e.g. 


(3.2)  ?R(z) 


But  set 


Fv‘» 


a*  j - bj  = 0,  j = 1,  . . . n 


35  ai;jZ  j . Then  if  >0  for  all  i and  6*TA  = bT. 


Note  that  for  this  choice 


,3.3,  *»•>■-?•?*  -5  (4) 

= p 6*  - p 8*  (tAz*)-  p 6*  - b V 


T Az  T * . . 

c e - b z = min  £ (z) 

../a\  a •>  n *T a - 


Hence  v(6 *)  = max  v(6),  6 a 0,  6TA  = b , and  min£(z)  = max  v(6). 

T T — 

Conversely,  suppose  6 a 0,  6 A = b has  a solution  6 > 0.  Then 


(3.  4)  £(z)  =2  ic.e  J aiiZJ  - IF  ZJ  a.  • z 1 

i ( 1 1 j lJ  JJ 


i 


But  the  function  of  one  variable  x,  with  c.,  6^  > 0, 

(3.  5)  j^j(x)  3 cjex  - 6jX  -*  + ® as  x -*  + ® 

Hence  if  zn  is  a sequence  for  which 

(3.6)  £(zn)  -Mnf  K(z) 

z 

we  cannot  have  jAzn  s £ a^z^  **  i " ' w^h  n -»  ®. 

Since  the  jAz11  are  thereby  bounded,  we  can  choose  a subsequence,  again 

denoted  zn,  so  that 

(3.  7)  jAzn  -*  a.,  all  i 

Applying  the  Farkas- Minkowski  Closure  Corollary  (FMC^)  [5J  we 

sjs 

conclude  there  exists  z such  that  ^Az  = a^,  all  i. 

But  then 

(3.8)  inf  £»(z)  = L (p(zn)  = c^ea  - 6^a  =fe(z  ') 
v n-*®  r 

i.e.  ^(z)  has  a minimum  at  z*. 

Thus  state  3 is  established: 

(2(z)  has  a minimum  iff  6 > 0,  6 A = b has  a solution  and  min  £(z)= 

max  v(6). 

By  exhaustion,  then,  state  2 is  established  : 

£(z)  has  only  an  infimum  iffZTSj  = 0 in  every  solution  to  6 * 0, 

T T 
6 A = b1  . 

To  obtain  the  further  properties  in  state  2,  we  note  first  that  since 
_T  T 

there  exists  6 a 0,  6 A = b , we  can  write 


£ (z)  = S [c.e(lAz*  - 64  (jAz)} 


(3.  81) 
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Now  suppose  £(zn)  -*  inf  ^(z)  and  that  (w.l.o.g. ) (^Az11)  -*  a^,  i e D, 
(rAzn)  -»  - ®,  r i D.  (Clearly  6r  = 0 for  r 4 D). 

By  the  FMC2  , there  exists  z such  that  (Az  = a4,  i e D,  rAz  si,  r 4 E 

Then  iA(zn-z)  -»  0,  i «D  and  rA  (z11  - z)  -*  - ®.  r i D.  By  the  FMC2,  there 

exists  z such  that  jAz  = 0,  i e D,  rAfc  s -1 

Then  iA(z  + pz)  = at  , i e D,  rA(z  + pz)  -»  - » as  p -*  * 

Thus  $ (z  + pz ) -*  inf  £ (z)  asp-*  ». 

But 

(3.82)  (z  + p2T)  = ,JD  [cieiAz  - (tAz)  > + r^D  cr  e + pz  } 
and 

(3.83)  $(z  +pz)  = inf^fz)  = { CielAz  - f>i  jAz)}  = SD  c^  A - b z 

= (?£)(z) 

Now  6 (z)  = £ C c,eiAz  - 6-(Az)}  has  a minimum  at  say  z , 

D ieD  1 1 

— - *p 

since  5;  > 0,  ieD  and  £6;  (;A)  = b 
i leD 

Thus  0 D(z)  s $D(z)  . If  ^ D<z>  < <*)  * then 

^ . $<z  + Pz>  = ^D(z)  < ^D(z)  = tof  6 (z),  a contradiction. 

So  £ D(z)  = min  £d<z)  = inf  £<z) 

Then  v(6)  = inf#z)  where  6*  = , i e I 

) 0 i *1 

Hence  v(6)  = max  v(6)  • ^ 

. . • * £ 

Thus  a 6V'  exists  and  by  uniqueness  0-6. 

The  further  properties  of  state  2 are  thereby  established,  e.g.  : 


(3.9) 


inf  tS(z)  = max  v (6)  = min  &d(z) 


As  noted  by  Charnes  and  Cooper  in  (l),the  duality  state  may  be 
characterized  by  means  of  the  linear  program: 


(3. 91) 


max  u 

M.e^  -6*0 

6TA=bT 

6^0 


State  1 corresponds  to  infeasibility,  state  2 to  u*=0,  state  3 to  |i*  > 0. 


4.  The  Duality  For  General  Distributions 

From  the  (two  real  variable)  result  in  (2)  and  (2.1).  we  can  write 
(4.1  ) -fi  (t)  ftt  r|^]  * c (t ) ex(t)-  6 (t )x(t), 
where  c(t)  > 0,  Mt)  i 0. 

Taking  p(t)  as  a non-negative  Radon-Stieltjes  measure, then,  assuming 
existence  of  the  integrals, 

(4.2)  v(fi)  ■ - du(t)  s ffc(t)ex(t)  - 6(t)x(t)jdu(t) 


(4.3) 


x(t)  - ^Aft,  s)z(s)dV(s)  , 

^6(t)  A(t,  s)  d p(t)  = b(s) 


where  v(s)  is  a Radon-Stieltjes  measure,  and  convergence  of  integrals  is 
assumed.  Then  (assuming  Fubini's  theorem  holds), 

(4.4)  v(  6)  * - js(t)<H  e^r  1 dn(t)  * £(z),  and 

£(*)  = j{c(t)  exp  [jA(t,  s)  z(s)  dv(s)  ] du(t)  - Jb(s)z(s)  d\/s) 
when  (4.  1 and  (4.  2)  hold. 

Clearly  with  6(t)  = fj(t),  c(t)  > 0 (except  on  a set  of  li-measure  zero) 
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corresponding  to  f2(t),  in  Kullback's  notation  [ 6 ],  we  have  here  exhibited 

the  Charnes -Cooper  dual  programs  for  the  case  of  general  distributions.  Note 
for  special  choice  of  v that  s takes  on  only  a finite  number  of  values  as  in 
Rockefellar 's  or  in  Ben-Tal's  and  Charnes'  work.  This  is  the  important 
case  in  statistical  estimation  of  a finite  number  of  constraints.  Thereby  #z) 
involves  only  an  unconstrained  finite  vector  z. 

5.  A Game -Theoretic  Connection 

We  can  also  connect  the  C2  -duality  with  a saddle -point  or  two-person 
zero-sum  game.  For,  let 

(5.1)  K(6,x)  sS  (cieXl  - Apq) 

be  the  payoff  to  the  "6 -player”  from  the  ”x-player”  Let  the 
respective  strategy  sets  A,  X be  defined  by 
A s {5;  6TA  = bT,  6*0) 

(5.2) 

X s [x:  x = Az) 

Clearly 

(5.3)  sup  v(6)  = sup-£6i0rj  ( 0 s sup  inf  K(6,  x) 

A A i ' 1 7 AX 

But 

(5.4)  sup  inf  K(6,  x)  « inf  sup  K(6,  x) 

AX  X A 

whenever  both  sides  have  meaning. 

Also, 

T Az  T 

inf  sup  K(  6,  x)  = inf  sup  (c  e - 6 Az) 

X A z A 

<5-5)  = inf  (cTeAz  - bTz),  when  A i , 

z 

= inf  £(z) 
z 

■ From  the  C2  duality  theory,  then,  sup  v(6)  = inf  £(z) 


■HHB 
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Thus, 

Theorem:  sup  inf  K(6,x)  = inf  sup  K(6)  whenever  A / 0 
AX  X A 

Or  more  sharply, 

max  inf  K(6,  x)  = inf  max  K(6,  x)  whenever  A i 0. 


6.  A More  General  Inequalities  Form 


form 
. 1) 


2 

Suppose  we  wish  to  consider  discrete  K L estimation  of  the 

max  v(6)  = ”6^07! 

subject  to  ta1  .IT 

o A - b 

6TA2  £ b2T 

6 2 0 


2) 


The  constraints  may  be  considered  equivalently  as 


= b1T 
6TA2+VT=  b2T 
6T,  yT2  0 


By  the  remark  on  c=0  in  $ 2, 
we  may  write  a duality  inequality  as 
.3)  -V  [^]  +o  s.S  (CieXl-  6.x.)  -£jyyr 


with  (6.  2)  holding  as  well  as 
.4)  yr  < 0,  i e Ir 

To  decouple  we  must  choose  representations  of  the  x.,yr  so  that  (6.2) 
represents  the  " T " and  so  that  Z>  comprehends  (6.4).  In  vector  form  we  want 
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It  suffices  to  have 

(6.6)  (6T,  yT) 


A1  A2 

0 I 
L-  J 


(biT.  b2T, 


or. 


(6. 


[xl  Ta1  A? 

[4 

krL°  1 j 

z2 

L J 

with  z2  s0 


The  dual  problems  are  now 
(I1)  max  v(6)  ■ 6^  [tfe] 

with  [&T,  YT|]  A1  A2  "J 


and 

(II’) 


6 , Y *0 

inf$  (z).  cTe  (A^1+A2^2> 
with  -z2  ^ 0 


til 


[b1T.  b**] 


,1  T 1 . 2T  2 

- D Z - D Z 


As  might  be  expected,  there  are  three  duality  states.  These  may 
be  established  as  above  and  characterized  by  the  solution  of  the  6 -constraint 
system  (6.1).  These  results  have  been  considered  by  a different  method 
by  F.  Palacios  (a  student  of  Charnes  ). 

The  duality  generalization  to  arbitrary  statistical  distribution  functions 
is,  of  course,  immediate,  as  follows. 


The  dual  problems  are 


(III')  sup  : v(f ! ) ■ - Jfj  flrf  f 1 \d  u 

T ' 

with  y f.(t)  A*  (t,  s)  d|i (t)  = b*(s),  s e Sj 

T 

J* f i<t ) A2  (t,  s)  du(t)  + g(s)  = b2(s),  s e S2 
T 

f 2 (t),  g(s)  2 0,  t e T,  s e S2  : 

(f2  is  given  and  greater  than  zero  a.e.  ~u) 

and 

(IV')  inf:  £(z)  5 ^ f g (t ) exp  ( ^ A*(t,  s)  z*(s)  dX  (s)  + J A2(t,  s)  z2(s)  dX2(s)]du 
T Sj  S2 

- ^b*(s)  z*(s)  dX*(s)  - $b2(s)  z2(s)  dX2  (s) 

Si  s2 

with  z2(s)  ^ 0,  s e S2 
where,  of  course,  the  duality  inequality 
(6.8)  v(fi)  * ^>(z)  holds  for  fj(t),  g(s),  z2(s) 

constrained  as  in  (III'),  (IV')  and  for  suitable  convergence  of  the 
integrals. 

7.  Concluding  Comments 

The  Charnes -Cooper  extremal  principle  herein  enunciated  is  more 
general  than  the  Fenchel-Rockefellar  construction  with  the  Legendre 
transformation(their  K(6,x)  concave  function rf  "6"  is  always  linear  in  5). 

As  illustrated  above,  the  principle's  simplicity  and  concreteness  should  make 
it  a useful  tool  for  the  practicing  analyst. 


A special  instance  of  the  game -theoretic  connection  is  being 
employed  by  Charnes,  Cooper  and  Schinnar  in  current  research  to  elucidate 
economic  phenomena  in  a cartel  economy. 

The  generalizations  to  functions  other  than  our(v6),^(z)  and  the 
precise  characterizations  of  the  "non-discrete"  dualities  are  also  to  be 
published  elsewhere.  These  are  important  for  problems  of  statistical 
inference  with  linear  inequality  constraints  since  the  dual  problem  is 
not  so  encumbered.  Further  they  apply  directly  to  problems  involving 
the  evolution  of  macroscopically  irreversible  systems  from  microscopically 
reversible  systems,  as  developed,  for  example,  by  B.  O.  Koopman  [ 7], 
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